> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/jaypopat/cf_ai_duet/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Integration

> Llama 3 integration for AI pair programming

## Overview

Duet uses Cloudflare's AI service with the Llama 3 8B Instruct model to provide intelligent pair programming assistance. The LLM generates responses based on conversation history and can output commands for sandbox execution.

## Model Configuration

The worker uses Cloudflare AI binding configured in `wrangler.toml`:

```toml theme={null}
[ai]
binding = "AI"
```

**Source:** `~/workspace/source/cf-worker/wrangler.toml:12-13`

The model is accessed via the `@cf/meta/llama-3-8b-instruct` identifier:

```typescript theme={null}
private async runAI(messages: AIMessage[]): Promise<string> {
  const result = await this.env.AI.run("@cf/meta/llama-3-8b-instruct", {
    messages,
  });
  return result.response?.trim() || "";
}
```

**Source:** `~/workspace/source/cf-worker/index.ts:122-127`

## Message Format

The AI expects messages in a specific format:

```typescript theme={null}
interface AIMessage {
  role: "system" | "user" | "assistant";
  content: string;
}
```

**Source:** `~/workspace/source/cf-worker/index.ts:84-87`

## System Prompt

Duet's AI behavior is defined by a system prompt that instructs it to be concise and use special tags for command execution:

```typescript theme={null}
const aiMessages: AIMessage[] = [
  {
    role: "system",
    content:
      "You are Duet, a concise pair-programming assistant. " +
      "You can run commands in a sandbox using <run>command</run> tags. " +
      "When asked to perform an action, briefly explain what you will do and wrap the exact shell command(s) in <run> tags. " +
      "Do NOT include predicted output in your response - just provide the explanation and command.",
  },
  // ...
];
```

**Source:** `~/workspace/source/cf-worker/index.ts:154-162`

This prompt establishes:

* **Identity**: Duet is a pair-programming assistant
* **Tone**: Concise and action-oriented
* **Command syntax**: Use `<run>command</run>` tags
* **Output handling**: Don't predict output, let sandbox provide it

## Conversation Context

The AI receives context from recent conversation history:

```typescript theme={null}
const aiMessages: AIMessage[] = [
  {
    role: "system",
    content: "You are Duet, a concise pair-programming assistant..."
  },
  ...this.state.messages.slice(-10).map<AIMessage>((m) => ({
    role: m.role === "agent" ? "assistant" : "user",
    content: m.text,
  })),
  { role: "user", content: userMsg.text },
];
```

**Source:** `~/workspace/source/cf-worker/index.ts:154-168`

The agent:

1. Includes the system prompt
2. Adds the last 10 messages from conversation history
3. Appends the current user message
4. Converts message roles ("agent" → "assistant", "user" → "user")

## Command Execution Flow

The AI can trigger sandbox commands using special tags:

### 1. AI Response with Commands

The AI wraps commands in `<run>` tags:

```
I'll check the directory contents.
<run>ls -la</run>
```

### 2. Command Extraction

The agent extracts commands using regex:

```typescript theme={null}
private async executeCommands(text: string, roomId: string): Promise<string> {
  const matches = Array.from(text.matchAll(/<run>([\s\S]*?)<\/run>/g));
  let result = text;

  for (const match of matches) {
    const cmd = match[1]?.trim();
    if (!cmd) {
      continue;
    }
    // ...
  }
}
```

**Source:** `~/workspace/source/cf-worker/index.ts:185-193`

### 3. Sandbox Execution

Each extracted command is executed in the room's sandbox:

```typescript theme={null}
try {
  const sandbox = getSandbox(this.env.Sandbox, `sandbox-${roomId}`);
  const { stderr, stdout } = await sandbox.exec(cmd);

  const summary =
    stdout.slice(0, 500) || stderr.slice(0, 500) || "[no output]";
  result += `\n\nOutput (${cmd}):\n${summary}`;
} catch (e) {
  const msg = e instanceof Error ? e.message : String(e);
  result += `\n\nError (${cmd}):\n${msg}`;
}
```

**Source:** `~/workspace/source/cf-worker/index.ts:195-205`

### 4. Output Appending

Command outputs are appended to the AI's response:

* First 500 characters of stdout or stderr
* Error messages if execution fails
* "\[no output]" if command produces nothing

### 5. Tag Removal

The `<run>` tags are stripped from the final response:

```typescript theme={null}
return result.replace(/<run>[\s\S]*?<\/run>/g, "").trim() || "";
```

**Source:** `~/workspace/source/cf-worker/index.ts:207`

## Complete Message Flow

```typescript theme={null}
private async handleMessage(roomId: string, rawBody: unknown): Promise<Response> {
  // 1. Validate request
  const parseResult = MessageRequestSchema.safeParse(rawBody);
  
  // 2. Create user message
  const userMsg: DuetMessage = {
    role: "user",
    userId: data.userId?.trim(),
    text: data.text.trim(),
    ts: Date.now(),
  };

  // 3. Build AI context
  const aiMessages: AIMessage[] = [
    { role: "system", content: "..." },
    ...this.state.messages.slice(-10).map(...),
    { role: "user", content: userMsg.text },
  ];

  // 4. Get AI response
  const text = await this.runAI(aiMessages);
  
  // 5. Execute any commands in response
  const textWithOutputs = await this.executeCommands(text, roomId);

  // 6. Create agent message
  const agentMsg: DuetMessage = {
    role: "agent",
    text: textWithOutputs,
    ts: Date.now(),
  };

  // 7. Update state
  const nextMessages = [...this.state.messages, userMsg, agentMsg].slice(-50);
  this.setState({ messages: nextMessages });

  // 8. Return response
  return Response.json({ reply: agentMsg.text, messages: nextMessages });
}
```

**Source:** `~/workspace/source/cf-worker/index.ts:129-183`

## API Request/Response

### Request Format

```json theme={null}
{
  "text": "list files in the current directory",
  "userId": "user-123"
}
```

Validated by:

```typescript theme={null}
const MessageRequestSchema = z.object({
  text: z.string().min(1, "Text cannot be empty"),
  userId: z.string().optional(),
});
```

**Source:** `~/workspace/source/cf-worker/index.ts:9-12`

### Response Format

```json theme={null}
{
  "reply": "I'll check the directory contents.\n\nOutput (ls -la):\ntotal 48\ndrwxr-xr-x  12 user  staff   384 Mar  1 10:00 .",
  "messages": [
    {
      "role": "user",
      "userId": "user-123",
      "text": "list files in the current directory",
      "ts": 1709294400000
    },
    {
      "role": "agent",
      "text": "I'll check the directory contents.\n\nOutput (ls -la):\n...",
      "ts": 1709294401000
    }
  ]
}
```

**Source:** `~/workspace/source/internal/ai/client.go:42-47`

## Next Steps

* [Sandboxes](/architecture/sandboxes) - Learn how commands are executed safely
* [Durable Objects](/architecture/durable-objects) - Understand conversation state management
* [Cloudflare Workers](/architecture/cloudflare-workers) - See the request routing layer